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BACKGROUND OF THE INVENTION 
10 1. Field Of The Invention 

The invention relates to interlaced video display, and 
more particularly to an image sensor for displaying 
interlaced video using optical summing. 

\^ 15 2. Description of the Related Art 

% In full color video applications, video cameras 

in typically employ the CMYG (Cyan, Magenta, Yellow, and Green) 

J« filter pattern for capturing full color video images. The 

iy CMYG filter scheme is preferred over the traditional RGB 

2 0 (Red, Green, Blue) primary color scheme because the CMYG 
C3 filters have higher transparency (which results in higher 

sensitivity) and because the use of CMYG colors simplifies 

CO post-capture image processing. Figure 1 illustrates a 

Q 

ijl typical CMYG filter pattern used in conventional video 

25 cameras. CMYG filter pattern 50 includes alternating rows 
of cyan/yellow filters (rows 51 and 53) and green/magenta 
filters (rows 52 and 54) . In the second row of 
green/magenta filters (row 54) , the positions of the green 
and magenta filters are switched so that a magenta filter is 

3 0 positioned below a cyan filter of the row above and a green 

filter is positioned below a yellow filter of the row above. 
The 4 by 4 block of filter pattern 50 is repeated to cover 
the entire array of pixel elements in the image sensor of 
the video camera. 
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video cameras output video data in an interlaced format 
because most display monitors display video data in an 
interlaced format. That is, video data are outputted by 
alternating between two data fields, an odd field and an 
5 even field, every 1/60^^ of a second. Video data associated 
with the odd rows of the image sensor are outputted in the 
odd field while video data associated with the even rows of 
the image sensor are outputted in the even field. The odd 
and even data fields are thus interlaced. Each of the odd 
10 and even data fields has a resolution that is half the 

vertical resolution of the image sensor. For example, if 
the image sensor has a resolution of 720 columns by 480 rows 
of pixel elements, each data field output a video image 
having a resolution of 720 by 240. 
H 15 To generate interlaced video data, the entire field of 

fS the image sensor is used. In operation, the image sensor 

y} captures a frame of image data and pixels values in adjacent 

f ri 

ffj rows of the image sensor are summed to generate the video 

W data for either the odd or the even data field. Then, the 

7' 2 0 image sensor captures another frame of image data and pixel 

p values in the alternate, adjacent rows of the image sensor 

ry are summed to generate the video data for the other data 

S field. 

ry Figure 2 illustrates the slimming operation for 

25 generating the interlaced video data in an image sensor 
overlaid with the CMYG filter pattern of Figure 1. The 
video data capture scheme is referred to as the CMYG 
interlaced capture scheme. Referring to Figure 2, image 
sensor 60 is illustrated as having 8 rows and 8 columns of 
3 0 pixel elements. To generate the odd field image, pixel data 
in adjacent pixel elements in rows 1 and 2, rows 3 and 4, 
rows 5 and 6 and rows 7 and 8 are summed, resulting in four 
rows of interlaced video data. For the even filed image, 
pixel data in adjacent pixel elements in rows 2 and 3, rows 
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4 and 5, and rows 6 and 7 are summed, resulting in another 
four rows of interlaced video data. 

Because video output signals are typically given in the 
yCbCr color space, post- capture image processing is 
5 performed to convert the pixel data captured in the CMYG 
color space to the FCbCr color space, (In the following 
description, the italic letter Y is used to denote the 
luminance channel in the YCbCr color space while the letter 
Y is used to denote the color yellow in the CMYG color 

10 space.) Color interpolation from CMYG to YCbCr is well 
known. Figure 3 illustrates the interpolation operation 
involved in converting the pixel data in CMYG color space to 
the YCbCr color space. 

Referring to Figure 3, the luminance channel Y is 

15 estimated by adding pixel values in adjacent columns of 
pixels. For example, for the odd field, the luminance 
channel Y for the first row is the sum of (C+G) and (Y+M) 
while the luminance channel Y in the second row is the sum 
of (C+M) and (Y+G) . The chrominance channels Cb and Cr are 

20 estimated by subtracting pixel values in adjacent columns of 
pixels, as shown in Figure 3. The chrominance channel Cr is 
the difference of (Y+M) and (C+G) while the chrominance 
channel Cb is the difference of (C+M) and (Y+G) . In this 
manner, the luminance channel Y is estimated at the 

25 intersection of every pixel while the chrominance channels 
Cb and Cr are estimated at every other pixel to form the 
desired color demosaic. The color interpolation for the 
even field operates in the same manner as in the odd field. 
The even field has a different color filter ordering but the 

30 same combination of rows. 

The use of the CMYG interlaced capture scheme has 
several advantages over the traditional RGB capture scheme. 
First, the CMYG color filters, being complementary colors 
and spanning a broader spectrum of light frequencies, are 

35 capable of capturing more light than RGB filters. Thus, the 
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sensitivity of the image sensor can be enhanced. Second, 
post-capture image processing is simplified. By using the 
CMYG filter pattern, video data output in the desired YCbCr 
color space can be obtained readily. Specifically, only one 
5 adder is needed to interpolate the luminance Y channel and 
only one subtracter is need to interpolate either of the 
chrominance channel Cb or Cr. Thus, the interpolation 
operation is computationally simple and can be performed at 
a very fast speed. High-speed interpolation is particularly 
10 important for video images as the output rate is one data 
field per 1/60^^ of a second. 

Finally, by summing pixel values of adjacent rows, the 
image sensor's sensitivity can be increased. This is true 
for image sensors where pixel data are measured in analog 

H 15 form and converted to digital values by circuits outside of 

the image sensor array, such as CCD or APS image sensors. 

U\ In an CCD image sensor, charge accumulates in each pixel 

element. The summing of the analog pixel values has the 

ty effect of increasing the photo- sensitive area of the pixel 

20 cell, albeit under two different color filters. By summing 

O the analog pixel values of adjacent pixels and then 

fsj digitizing the summed value, inaccuracy due to quantization 

ffl noise can be avoided. 

n\ However, when the image sensor m the video camera is 

25 one in which pixel-level analog-to-digital conversion (ADC) 
is performed, the summing operation has to be performed 
using the digitized pixel values. Summing digitized pixel 

values has several disadvantages. First, the summing 
operation will become more computationally complex. The 
30 digital values have to be added using a data processor. 
Second, because of quantization noise in the digitized 
signals, the effective sensitivity of the image sensor is 
decreased. 



-4- 



Therefore, it is desirable to provide an alternate 
implementation of the CMYG interlace capture scheme in an 
image sensor to avoid the aforementioned disadvantages. 

SUMMARY OF THE INVENTION 

According to one embodiment of the present invention, 
an image sensor includes a two-dimensional array of pixel 
elements where the array of pixel elements outputting pixel 
data representing an image of a scene, and a two-dimensional 
array of selectively transmissive filters superimposed on 
the two-dimensional array of pixel elements, whereby each 
pixel element in the array of pixel elements is disposed to 
capture a first and a second color spectra of visible light. 

In operation, light intensity values for two different 
color spectra are optically summed at the pixel level, 
providing pixel values that are suitable for use in 
interlaced video display. The image sensor of the present 
invention has applications in a video camera employing the 
CMYG (cyan, magenta, yellow, and green) interlaced capture 
scheme for capturing full color video images. 

In one embodiment, the two-dimensional array of pixel 
elements is a sensor array of digital pixels where each of 
the digital pixels outputs digital signals as pixel data. 
In another embodiment, the pixel elements in the array of 
pixel elements generates analog signals as pixel data. 

According to another aspect of the present invention, a 
method for generating electrical signals representing an 
image in an image sensor where the image sensor includes an 
array of pixel elements overlaid with an array of 
selectively transmissive color filters includes the step of 
generating at each pixel element pixel data indicative of 
the light intensity impinging on the pixel element, where 
the pixel data is a sum of the light intensity of a first 
color spectrum and a second color spectrum of visible light. 



The present invention is better understood upon 
consideration of the detailed description below and the 
accompanying drawings. 



5 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates a typical CMYG filter pattern used 
in conventional video cameras. 

Figure 2 illustrates the summing operation for 
generating the interlaced video data in an image sensor 
10 overlaid with the CMYG filter pattern of Figure 1. 

Figure 3 illustrates the interpolation operation 
involved in converting the pixel data in CMYG color space to 
the YCbCr color space. 

Figure 4 illustrates an image sensor according to one 
15 embodiment of the present invention. 
!;S Figure 5 is an enlarged view of a pixel element 

in overlaid with two transmissive color filters according to 

J-J; one embodiment of the present invention. 

W Figure 6 is a cross -sectional view of a pixel element 

20 and the associated color filters according to one embodiment 
Q of the present invention. 

as: 

fu Figure 7 illustrates another embodiment of the present 

K invention where pixel-level optical summing is accomplished 

using a color filter array including filter cells. 
25 Figures 8a to 8d are curves illustrating the spectral 

responses which can be used to define the four composite 
filters for use in the CMYG interlaced capture scheme. 

Figure 9 is a cross -sectional view of a pixel element 
and an associated composite color filter according to one 
3 0 embodiment of the present invention. 

Figure 10 illustrates the digital pixel sensor 
architecture used in the '42 5 patent which may be used to 
practice the apparatus and method of the present invention. 
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In the present disclosure, like objects which appear in 
more than one figure are provided with like reference 
numerals . 



5 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In accordance with the principles of the present 
invention, an image sensor includes an array of pixel 
elements overlaid with an array of selectively transmissive 
color filters so that each pixel element can sense two or 

10 more color spectra of visible light. In this manner, light 
intensity values for two different color spectra are 
optically summed at the pixel level, providing pixel values 
that are suitable for use in interlaced video display. In 
one embodiment, the array of selectively transmissive 

15 filters is shifted so that each pixel element is overlaid 
with two different transmissive color filters. Each pixel 
element is thus disposed to capture incident light of two 
colors. In another embodiment, each filter cell of the 
array of transmissive filters is a composite filter capable 

20 of transmitting transmit light in two or more color spectra. 
Each composite filter cell overlies a pixel element so that 
each pixel element operate to capture two color spectra of 
visible light. 

Figure 4 illustrates an image sensor 100 according to 

25 one embodiment of the present invention- Image sensor 100 
includes a two-dimensional array of pixel elements 102 and a 
two-dimensional array of selectively transmissive filters, 
also referred to as a color filter array 104. In the 
present embodiment, color filter array 104 is in offset 

30 registration with the array of pixel elements so that each 
color filter overlies two adjacent pixel elements in a 
column. Referring to Figure 4, color filter array 104 is 
shifted vertically by half a pixel so that each color filter 
overlies a portion of the active area of two adjacent pixel 

35 elements. Alternately, each pixel element can be viewed as 
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being overlaid by two color filters. For example, a pixel 
element 108 is overlaid with a magenta filter 109 on the top 
and a yellow filter 110 on the bottom. By superimposing two 
color filters on each pixel element, optically summing of 
5 the light intensity values from the two different color 
filters can be accomplished. 

Figure 5 is an enlarged view of pixel element 108 
overlaid with two transmissive color filters 109 and 110, 
according to one embodiment of the present invention. In 

10 the present embodiment, color filter array 104 is shifted by 
half a pixel so that each color filter overlies half of 
pixel element 108. Figure 6 is a cross-sectional view of a 
pixel element and the associated color filters according to 
one embodiment of the present invention. In Figure 6, pixel 

15 element 108 is implemented as a photodiode including a 

photogate 2 02 which accumulate charges in the substrate area 
212. Pixel element 108 may also include other components, 
such as a control gate, a sense node 210 and a reset 



transistor 216. Photogate 202 is overlaid with two color 

Co 

.„ 20 filters 109 and 110. In operation, when incident light 

impinges upon pixel element 108, filter 109 will transmit 
ry light in the frequency range of the magenta color spectrum 

and filter 110 will transmit light in the frequency range of 
fu the yellow color spectrum. Photogate 202 collects charges 

25 representative of the light intensity values of the sum of 
the magenta color and the yellow color. In this manner, 
pixel -level optical summing of the light intensity values of 
the magenta color and the yellow color is accomplished. 
Figure 4 illustrate one embodiment of the present 
30 invention where pixel-level optical summing is realized by 
shifting the alignment of the color filter array with 
respect to the array of pixel elements. Other embodiments 
for realizing pixel-level optical summing are possible. 
Figure 7 illustrates another embodiment of the present 
35 invention where pixel-level optical summing is accomplished 
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using a color filter array including filter cells. In the 
embodiment shown in Figure 7, instead of offsetting the 
color filter array to achieve pixel -level optical summing, 
image sensor 10 0 includes a color filter array having a 
5 predefined filter pattern. Specifically, the color filter 
array is made up of an array of filter cells where each 
filter cell incorporates two filter types. 

Referring to Figure 7, a pixel element 112 in the array of 
pixel elements is overlaid by a filter cell 114 in the color 
10 filter array. Filter cell 114 includes two types of 

filters: a yellow filter 115 and a green filter 116. Filter 
cell 114 is divided into four quadrants and each of filter 
115 and 116 occupies two quadrants of the filter cell. In 
the present embodiment, yellow filter 115 occupies two 
H 15 quadrants that are diagonal to each other (quadrants 115a 

7^ and 115b) and green filter 116 occupies the two remaining 

in quadrants that are also diagonal to each other (quadrants 

CO 

116a and 116b) . Filter cells incorporating other 
W combinations of filter types, such as cyan and magenta, are 

l^' 20 formed so as to create the desired color mosaic for the 

"^^^ color filter array. 

fu The use of filter cells in the color filter array of 

';3=f image sensor 100 to incorporate two filter types at each 

fil pixel location has advantages in that the alignment of the 

25 color filter array to the array of pixel elements is less 
critical. When the color filter array is shifted by half 
pixel to achieve optical summing (Figure 4) , the alignment 
of the color filter array to the array of pixel elements 
must be precisely done to ensure that each pixel location is 
30 overlaid with the correct proportion of filter material. 
However, when a color filter array including filter cells 
such as filter cell 112 (Figure 7) is used, the color filter 
array can be readily aligned with the array of pixel 
elements and the precision of alignment becomes less 
35 critical. 
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According to yet another embodiment of the present 
invention, the image sensor of the present invention 
includes a color filter array made up of composite filter 
cells. In the present description, a composite filter or a 
composite filter cell is a filter designed to transmit light 
in two or more color spectra. In other words, a composite 
filter has the same spectral response as the combination of 
the spectral response of two different color filters. 

In order to realize the CMYG filter pattern for 
facilitating CMYG interlaced capture scheme, four composite 
filters are required. Specifically, a C+G composite filter, 
a C+M composite filter, a Y+M composite filter and a Y+G 
composite filter are required. The desired spectral 
response for a composite filter can be found by adding the 
spectral response of the individual colors and normalizing 
the resultant response to one. Figures 8a to 8d are curves 
illustrating the spectral responses which can be used to 
define the four composite filters for use in the CMYG 
interlaced capture scheme. Figure 8a illustrates the 
spectral response for a C+G composite filter. Figure 8b 
illustrates the spectral response for a C+M composite 
filter. Figure 8c illustrates the spectral response for a 
Y+M composite filter. Figure 8d illustrates the spectral 
response for a Y+G composite filter. The composite filters 
can be manufactured by applying the correct amount of 
chemicals to obtain the desired spectral response for each 
composite filter. 

Having defined the appropriate spectral response for 
each of the composite filter types, a color filter array can 
be formed by arranging the composite filter cells to form 
the desired CMYG filter pattern. Each filter cell is in 
registration with each pixel element in the image sensor. 
Figure 9 is a cross-sectional view of a pixel element and an 
associated composite color filter according to one 
embodiment of the present invention. In Figure 9, the pixel 
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element 3 08 is implemented as a photodiode in the same 
manner as the photodiode in Figure 6, Like elements in 
Figures 6 and 9 are given like reference numerals. A 
composite filter cell 310 having a spectral response of C+G 
overlies photogate 202. Thus, when pixel element 308 is 
exposed to incident light, photogate 202 accumulates charges 
that are indicative of the light intensity value of the cyan 
and the green color spectra. In this manner, optical 
summing of the light intensity of the cyan color and the 
green color is accomplished through the use of a composite 
filter. 

The use of a composite filter in the image sensor of 
the present invention to facilitate optical summing has 
several advantages. First, the color filter array can be 
readily aligned with the array of pixel elements and precise 
alignment of filter cells to the pixel elements is not 
critical. Second, by providing a composite filter instead 
of using two or more adjoining filters, any reflection from 
the internal interface of the adjoining filters can be 
entirely avoided. 

The image sensor of the present invention is an area 
image sensor that can be implemented as a charged-coupled 
device (CCD) or a Complementary Metal-Oxide Semiconductor 
(CMOS) image sensor. CMOS image sensors may include active 
pixel sensors (APS) and digital pixel sensors (DPS) . The 
image sensor of the present invention can be implemented 
using any image sensors, presently available or to be 
developed. Typically, the image sensor includes a two- 
dimensional array of light detecting elements, also called 
photodetectors . 

CCD and APS image sensors are well known in the art. 
The operation of CCD image sensors and APS image sensors are 
similar in that both types of image sensors operate to 
produce an analog output signal at each pixel. If the image 
sensor of the present invention is an analog image sensor 
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such as a CCD image sensor, then the image sensor may 
include additional circuitry such as a readout and control 
circuits including an analog-to-digital converter (ADC) for 
converting the analog image signals into digital signals. 
5 In the present embodiment, the image sensor of the 

present invention is a digital pixel sensor (DPS) . A CMOS 
image sensor with pixel level analog-to-digital conversion 
is described in U.S. Patent No. 5,461,425 of B. Fowler et 
al. (the '425 patent), which reference is incoirporated 

10 herein by reference in its entirety. Figure 10 duplicates 
Figure 1 of the M25 patent and illustrates the digital 
pixel sensor architecture used in the '425 patent which may 
be used to practice the apparatus and method of the present 
invention. In Figure 10, image sensor 10 is referred to 

15 herein as a digital pixel sensor (DPS) which provides a 
digital output signal at each pixel element (e.g. element 
15) representing the light intensity detected by that pixel 
element. The combination of a photodetector and an analog- 
to-digital (A/D) converter in an area image sensor affords 

20 many useful features that a conventional image sensor could 
not readily accomplish. 

In the present description, a digital pixel sensor 
(DPS) array or a sensor array refers to a digital image 
sensor having an array of photodetectors where each 

25 photodetector produces a digital output signal. The pixels 
of a DPS array are sometimes referred to as a ''sensor pixel" 
or a "sensor element" or a "digital pixel," which terms are 
used to indicate that the output of each pixel is in digital 
form, and is distinguishable from a conventional 

30 photodetector which includes a photodetector and produces an 
analog signal. Depending on an exact implementation, an 
analog-to-digital circuit may be included in a sensor 
element or a portion from several neighboring sensor 
elements forms a circuit that performs an analog-to-digital 

35 conversion. In any case, each of the sensor elements is 
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capable of producing digital output signals of high 
fidelity. The digital output signals of a DPS array have 
many advantages over the conventional analog signals, one of 
which is that the digital signals can be read out at a much 
5 higher speed. Of course, other schemes for implementing a 
pixel level A/D conversion in an area image sensor may also 
be used in the image sensor of the present invention. 

In the present embodiment, the image sensor may include 
a data memory for storing the digital output signals from 

10 the sensor array. The data memory can be built on the same 
integrated circuit as the image sensor or it can be built on 
a separate integrated circuit from the image sensor. In one 
embodiment, the data memory is an integrated on-chip memory 
of the image sensor and has the capacity for storing at 

15 least one frame of image data from the sensor array. U.S. 
patent application No. 09/567,638 describes an integrated 
digital pixel sensor (DPS) with an on-chip memory for 
storing at least a frame of image data. The incorporation 
of an on-chip memory in an image sensor alleviates the data 

20 transmission bottleneck problem associated with the use of 
an off -chip memory for storage of the pixel data. The 
aforementioned patent application is incorporated herein by 
reference in its entirety. 

The image sensor of the present invention employing 

25 optical summing in the pixel elements provides advantages 

over the conventional image sensor for generating interlaced 
video data. 

First, when the image sensor of the present invention 
is applied in a video camera, interlaced video data can be 

30 readily generated. Because each row of pixel elements in 
the image sensor includes the optical sum of the pixel 
values of two vertically adjacent color filters in the CMYG 
pattern, the odd field or the even field of the interlaced 
video data can be generated by reading out pixel values from 

35 every other row of image sensor 100. 
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For instance, referring to Figure 4, to generate the 
odd field, pixel data from the odd rows (i.e., rows 1, 3 and 
5) of image sensor 10 0 are read out. To generate the even 
field, pixel data from the even rows (i.e., rows 2, 4 and 6) 
5 of image sensor 100 are read out. In the present 

description, the terms ''odd rows" and ''even rows" are used 
to refer to a first set of alternate rows and a second set 
of alternate rows in the image sensor different than the 
first set. The use of the terms "odd rows" and "even rows" 
10 is not intended to limit the set of alternate rows to those 
rows that are numerically numbered with odd numbers or even 
numbers . 

Second, by optically summing pixel data of two 
different colors at each pixel location, an interlaced frame 

15 is created by every other rows in the image sensor. Thus, 
to generate the odd field or the even field images, only 
half the rows of the image sensor are read out. Thus, while 
half the rows are being read out to generate the video data 
for one data field (the odd or the even field) , the other 

20 half of the rows can continue to be exposed to the incident 
light, thereby increasing the integration time. For 
instances, the typical interlaced data rate for video 
display is 1/60*^^ of a second. Thus, in a conventional 
image sensor, the integration time for each data field is 

25 1/60^^ of a second. When optical summing in accordance with 
the present invention is applied, each field can be exposed 
to a full 1/3 0^^ of a second, effectively doubling the 
integration time and the sensitivity of the image sensor. 
Third, the optical summing technique of the present 

30 invention is particularly useful when applied in a DPS image 
sensor. When the CMYG interlaced capture scheme is applied 
to a DPS image sensor, pixel values of two adjacent rows are 
digitized at each pixel location and then the digital values 
are summed. Summing digitized pixel values can lead to 

35 inaccuracy due to quantization noise. In accordance with 
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the present invention, optical summing of pixel values is 
performed during the light integration process. Therefore, 
when the DPS image sensor digitizes the pixel value at each 
pixel location, the pixel value is already a summed value of 
two adjacent filter colors. By applying the optical summing 
technique of the present invention in a digital pixel 
sensor, the accuracy and sensitivity of the image sensor can 
be enhanced. 

In the image sensor of the present invention, the 
spatial sampling rate is different than in the traditional 
interlaced data capture scheme. In the image sensor of the 
present invention, only half the sensor area is used for a 
given data field. In the event that this spatial sampling 
creates aliasing artifacts, a simple blur function can be 
added to the processing to eliminate the imaging artifacts. 

The above detailed descriptions are provided to 
illustrate specific embodiments of the present invention and 
are not intended to be limiting. Numerous modifications and 
variations within the scope of the present invention are 
possible. For example, in the above descriptions, the 
alternate rows of the image sensor are referred to as odd 
rows and even rows for ease of reference. The notations of 
odd rows and even rows do not refer to the numeric numbering 
of the rows. The present invention is defined by the 
appended claims. 
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